Virtual reality and interactive and immersive planning for the assistance to manipulation or navigation

In industry, whereas the economic competition increases, up-to-date industrial products are more and more integrated and the tasks related to their lifecycle (assembly, maintenance, disassembly…) have to be performed under sometimes very strong geometric constraints. In the context of Industry 4.0 (Factory of the Future) and PLM (Product Lifecycle Management), industrial companies therefore express the needs to validate these tasks from design stage on, in order to be able to update the design of these products (before manufacturing the physical prototypes) if needed. Such an approach allows to reduce development time and cost, to detect errors as early as possible, and to target more environment friendly development processes. When simulating such complex scenarios, it is necessary to deal with the relative positioning or to the movement of objects and of resources (machines, robots, human operators) that manipulate them. A key issue is then to find a path, a trajectory, a movement to show the feasibility of scenarios and simulate what the execution of a task will be. Our works deal with the assistance to the simulation and validation of such complex scenarios in Virtual Reality. We present the original scientific approach on which these works are based: the joint use of motion planning and VR techniques to validate the feasibility of the movement for the simulated scenarios in an interactive and immersive way, with visuo-haptic guidance to the human operator in the loop. The initial approach was based on the use of purely geometric models. In order to improve the relevance of the assistance and the modalities of interaction and control sharing (authority sharing and intents detection) between the VR platform and the human operator, we then considered higher abstraction level (topological and semantic) data than the purely geometrical data traditionally used. Finally, for a better, task- or trade-oriented assistance, and in order to move from the "virtual experience" to the "trade-oriented experience", our work now targets the development of joint, interactive and immersive task and path planning strategies

Interactive multimodal Path Planning in immersion

Recent studies have defined interactive path plan- ners for simulations involving a human operator. Such path planners enable a human operator to share control with an automatic planner and are based on Robotics and Virtual Reality (VR) methods. This paper proposes a novel architecture for this interactive planner. It enhances interaction with the user by adding topological and semantic representations to the purely geometric model traditionally used

A multi-layer approach of interactive path planning for assisted manipulation in virtual reality

This work considers Virtual Reality (VR) applications dealing with objects manipulation (such as industrial product assembly, disassembly or maintenance simulation). For such applications, the operator performing the simulation can be assisted by path planning techniques from the robotics research field. A novel automatic path planner involving geometrical, topological and semantic information of the environment is proposed for the guidance of the user through a haptic device. The interaction allows on one hand, the automatic path planner providing assistance to the human operator, and on the other hand, the human operator to reset the whole planning process suggesting a better suited path. Control sharing techniques are used to improve the assisted manipulation ergonomics by dynamically balancing the automatic path planner authority according to the operator involvement in the task, and by predicting user’s intent to integrate it as early as possible in the planning process

A hierarchic approach for path planning in virtual reality.

This work considers path-planning processes for manipu- lation tasks such as assembly, maintenance or disassem- bly in a virtual reality (VR) context. The approach con- sists in providing a collaborative system associating a user immersed in VR and an automatic path planning process. It is based on semantic, topological and geometric representations of the environment and the planning process is split in two phases: coarse and fine planning. The automatic planner suggests a path to the user and guides him trough a haptic device. The user can escape from the proposed solution if he wants to explore a possible better way. In this case, the interactive system detects the users intention and computes in real-time a new path starting from the users guess. Experiments illustrate the different aspects of the approach: multi-representation of the en- vironment, path planning process, users intent prediction and control sharing

Multi-layer path planning control for the simulation of manipulation tasks : involving semantics and topology

The industrial and research communities show increasing interest in using automatic path planning techniques for the simulation of manipulation tasks. Automatic path planning, largely explored by the robotics community over the past 30 years, computes the trajectories of robots or manipulated parts. However, as techniques developed so far use mostly purely (and large) geometric models, they may fail, produce a trajectory of little relevance, or lead to very high computation times, when facing complex or very constrained environments. Involving higher abstraction level information should lead to better relevance of the simulation. In this paper, we propose a novel path planning technique relying on an original multi-layer environment model containing geometrical, topological and semantic layers. A first coarse planning step at the topological and semantic layers and a fine planning step at the local and semantically characterized geometrical layer form the path planning process. Experimental full-scale results show increased control on the planning process, leading to much lower computation times and increased relevance of the computed trajectory

Path planning control using high abstraction level environment model and industrial taskoriented knowledge

In order to face an increasing economic competition, industrial manufacturers wish to reduce the time and cost of product development. Furthermore, up-to-date products are more and more integrated, and must be assembled, disassembled or maintained under potentially very strong geometric constraints. In the context of Industry 4.0, manufacturers are therefore expressing the desire to validate all the tasks related to their products lifecycles, from design stage on, by simulation using a digital mock-up, and before building the physical prototypes. A key issue is then to find a trajectory, a movement, to show the feasibility of the simulated scenarios. Automatic path planning algorithms, developed by the robotics community from the 1980s on, have been widely used for this purpose. In this paper, we intend to improve the relevance of the trajectories proposed by such algorithms and the associated computation times. To do so, we consider: a) the use of path planning algorithms or of combinations of these; b) the involvement for the environment modelling of data with a higher abstraction level than the purely geometric data traditionally used; and c) the representation of the knowledge related to the task to be performed by using ontologies. The approaches developed and associated improvements of the state of the art are validated experimentally through the simulation of highly geometrically constrained manipulation tasks

Using virtual reality and 3D industrial numerical models for immersive interactive checklists

At the different stages of the PLM, companies develop numerous checklist-based procedures involving prototype inspection and testing. Besides, techniques from CAD, 3D imaging, animation and virtual reality now form a mature set of tools for industrial applications. The work presented in this article develops a unique framework for immersive checklist-based project reviews that applies to all steps of the PLM. It combines immersive navigation in the checklist, virtual experiments when needed and multimedia update of the checklist. It provides a generic tool, independent of the considered checklist, relies on the integration of various VR tools and concepts, in a modular way, and uses an original gesture recognition. Feasibility experiments are presented, validating the benefits of the approach

Semantic coupling of path planning and a primitive action of a task plan for the simulation of manipulation tasks in a virtual 3D environment

This work deals with the simulation of complex manipulation tasks in virtual environments. Validating such complex tasks, possibly to be performed under strong geometric constraints, requires considering task and path planning jointly. The contribution of this work focuses on using task-related information at the path planning level. We propose an ontology-based approach to a) model the 3D environment where the simulated task is executed, based on an original multi-level environment model involving higher abstraction level data than the purely geometric models traditionally used, and b) automatically define path planning queries for the primitive ctions of a task plan, together with task-related geometric constraints on these queries. This approach allows the improvement of the state of the art from two points of view. First, our joint task and path planning approach allows the improvement of path planning through better semantic control of the path planning process. Second, if compared to hard-coded geometric constraints, the proposed ontology-based approach introduces a more flexible ay of defining geometric constraints through an inference process, and can be adapted to different applications of manipulation tasks

An ontology-based approach towards coupling task and path planning for the simulation of manipulation tasks

Simulating complex industrial manipulation tasks (e.g., assembly, disassembly and maintenance tasks) under strong geometric constraints in a virtual environment, requires the joint usage of task and path planning, not only to compute a sequence of primitive actions (i.e., a task plan) at task planning level to identify the order to manipulate different objects (e.g., assembly order), but also to generate and validate motions for each of these primitive actions in a virtual environment by computing valid collision-free paths for these actions at path planning level. Although task and path planning have been respectively welly discussed by artificial intelligence and robotic domain, the link between them still remains an open issue, in particular because path planning for a primitive action often uses purely geometric data. This purely geometric path planning suffers from the classical failures (i.e., high-possibility of failure, high processing time and low path relevance) of automated path planning techniques when dealing with complex geometric models. Thus, it can possibly lead to high computational time of the joint task and path planning process and can probably produce a poor implementation of a task plan. Instead of geometric data, involving higher abstraction level information related to a task to be performed in the path planning of a primitive action could lead to a better relevance of simulations. In this work, we propose an ontology-based approach to generate a specific path planning query for a primitive action, using a well-structured task-oriented knowledge model. This specific path planning query aims at obtaining an increased control on the path planning process of the targeted primitive action

A multi-layer approach for interactive path planning control.

This work considers path-planning processes for manipulation tasks such as assembly, maintenance or disassembly in a Virtual Reality (VR) context. The approach consists in providing a collaborative system associating a user immersed in VR and an automatic path planning process. It is based on semantic, topological and geometric representations of the environment and the planning process is split in two phases: coarse and fine planning. The automatic planner suggests a path to the user and guides him trough a haptic device. The user can escape from the proposed solution if he wants to explore a possible better way. In this case, the interactive system detects the user’s intention in real-time and computes a new path starting from the user’s guess. Experiments illustrate the different aspects of the approach: multi-representation of the environment, path planning process, user’s intent prediction and control sharing